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Protocol consists of a state machine and a set of monitors which execute 
once every local oscillator tick. 


Monitor: 

case (incoming message from the corresponding node) 

{Resync: 

if invaiidSync( ) 

Invalidate the message ' 

else 

Validate and store the message 

Other: 

Do nothing 
} // case 

ConsumeMessage( ) 


503 

504 

Node: 

case (state of the node) 

{Restore 

Maintain: 

if TimeOutRestore( ) 
Reset StateTimer, 
Go to Maintain state. 

if TimeOutMaintain( ) or Retry( ) 
Transmit Sync message, 
Reset StateTimer, 

Go to Restore state. 

else 

if TransitoryConditionsMet( ) 
Reset StateTimer, 

Go to Maintain state. 

else 

if TimeOutGammaTimer( ) 
if (State Timer = fAp re cision/Y^) 
Reset Local Timer., 

Stay in Maintain state. 

else 

Stay in Restore state. 

else 

Stay in Maintain state. 


} // case 



Fig. 5 
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SELF-STABILIZING 

BYZANTINE-FAULT-TOLERANT CLOCK 
SYNCHRONIZATION SYSTEM AND 
METHOD 

5 

CROSS-REFERENCE TO RELATED 
APPLICATIONS 

This application claims the benefit of the filing date of U.S. 
Provisional Patent Application Ser. No. 61/056,537 filed May 10 
28, 2008, the entire disclosure of which is incorporated herein 
by reference. 

ORIGIN OF THE INVENTION 

15 

The invention was made in part by employees of the United 
States Government and may be manufactured and used by or 
for the Government of the United States of America for gov- 
ernmental purposes without the payment of any royalties 2 o 
thereon or therefor. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 25 

The present invention is in the field of real-time systems 

and more particularly concerns systems and methods for syn- 
chronizing clocks among a plurality of distributed nodes in a 
manner that is capable of reliably self-stabilizing even in the 
presence of nodes exhibiting arbitrary Byzantine fault-prone 30 
behavior. 

2. Description of the Related Art 

A major problem in operating with any distributed system 
is establishing a consistent global view of the system from the 
local perspective of the participants. A basic aspect of arriving 35 
at such consistency is the ability to synchronize clocks, 
Numerous methods have been devised for clock synchroni- 
zation, and for achieving convergence in resynchronization. 
The worst case scenario for synchronization is where the 
nodes to be synchronized are subject to “Byzantine” faults — 40 
that is, where distributed systems experience arbitrary and/or 
malicious faults during the execution of algorithms, includ- 
ing, among others, “send and omission failures”. See gener- 
ally H. Kopetz, “Real-Time Systems, Design Principles for 
Distributed embedded Applications” (Kluwer Academic 45 
Publishers, 1997) (hereinafter “Kopetz 1997”). Known sys- 
tems have not been able to guarantee convergence determin- 
istically, scalably, and in a self-stabilizing manner in the 
presence of Byzantine faults, without limiting assumptions 
about initial states, use of a central clock, or relying on an 50 
externally-generated pulse system. 

SUMMARY OF THE INVENTION 

It is an object of the invention to provide systems and 55 
methods for synchronizing distributed clocks that self-stabi- 
lize from any state; do not rely on any assumptions about the 
initial state of the clocks, and do not require a central clock or 
an externally-generated pulse system, but which converge 
deterministically; are scalable; and/or self- stabilize in a short 60 
amount of time, even with the inclusion of nodes exhibiting 
Byzantine faults. 

It is a further object of the invention to provide systems and 
methods for rapid Byzantine-fault-tolerant synchronization 
that tolerates bursts of transient failures, and deterministically 65 
converges with a linear convergence time with respect to the 
self-stabilization period. 


2 

It is another object of the invention to provide systems and 
methods for rapid Byzantine-fault-tolerant synchronization 
that are scalable with respect to the fundamental parameters 
of number of nodes (K), minimum event-response delay (D) 
and network imprecision (d). 

In at least one embodiment, the foregoing objects are 
achieved by the use, in systems and methods for distributed 
clock synchronization, of a protocol comprising a state 
machine and a set of monitors that execute once every local 
oscillator tick. This protocol is independent of application- 
specific requirements and, thus, is focused only on clock 
synchronization of a system in the presence of Byzantine 
faults and after the cause of transient faults has dissipated. 
Instances of the protocol are proven to tolerate bursts of 
transient failures and deterministically converge with a linear 
convergence time with respect to the synchronization period 
as predicted. This protocol does not rely on any assumptions 
about the initial state of the system and no assumptions are 
made about the internal status of the nodes, the monitors, and 
the system as a whole) thus making the weakest assumptions 
and, therefore, producing the strongest results. All timing 
measures of variables are based on the node’s local clock and 
thus no central clock or externally generated pulse is used. 
The Byzantine faulty behavior modeled here is a node with 
arbitrary and/or malicious behavior. The Byzantine faulty 
node is allowed to influence other nodes at every clock tick 
and at all times. The only constraint is that the interactions are 
restricted to defined interfaces. 

Other aspects and advantages of the invention will be 
apparent from the accompanying drawings, and the detailed 
description that follows. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present inven- 
tion and the advantages thereof, reference is now made to the 
following description taken in conjunction with the accom- 
panying drawings, wherein like reference numerals represent 
like parts, in which: 

FIG. 1 is a timeline depiction of event response delay and 
network imprecision. 

FIG. 2 is a block diagram for a node in accordance with one 
embodiment of the invention, showing its monitors and state 
machine. 

FIG. 3 is a state machine diagram of an exemplary node 
state machine. 

FIG. 4 is a timeline depiction of the activities of a good 
node during steady state. 

FIG. 5 shows three inter-related blocks of pseudocode, 
representing implementations of the components of a self- 
stabilization protocol in accordance with one embodiment of 
the invention. 

FIG. 6 is a flow chart showing the interaction of coarse and 
fine level protocols in another embodiment of the invention. 

DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENTS 

The following is a detailed description of certain embodi- 
ments of the invention chosen to provide illustrative examples 
of how it may advantageously be implemented. The scope of 
the invention is not limited to the specific embodiments 
described, nor is it limited by any specific implementation, 
composition, embodiment or characterization depicted in the 
accompanying drawings or stated or described in the inven- 
tion summary or the abstract. In addition, it should be noted 
that this disclosure describes a number of methods that each 
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comprise a plurality of steps. Nothing contained in this writ- 
ten description should be understood to imply any necessary 
order of steps in such methods, other than as specified by 
express claim language. 

1 . System Description 

The present disclosure is primarily directed at one advan- 
tageous embodiment, with a number of suggested alternatives 
and extensions. In this advantageous embodiment, the under- 
lying topology considered is a network of K>3F+1 nodes that 
communicate by exchanging messages through a set of com- 
munication channels. A maximum of F Byzantine faulty 
nodes are assumed to be present in the system, where F>0. 
The Byzantine nodes may be modeled as nodes with arbitrary 
and/or malicious behavior that may influence other nodes at 
every clock tick and at all times, The communication chan- 
nels are assumed to connect a set of source nodes to a set of 
destination nodes such that the source of a given message is 
distinctly identifiable from other sources of messages, The 
minimum number of good nodes in the system, G, is given by 
G=K-F nodes. Let K G represent the set of good nodes. The 
nodes communicate with each other by exchanging broadcast 
messages. Broadcast of a message to all other nodes is real- 
ized by transmitting the message to all other nodes at the same 
time. The source of a message is assumed to be uniquely 
identifiable. The communication network does not guarantee 
any relative order of arrival of a broadcast message at the 
receiving nodes. To paraphrase Kopetz [Kopetz 1997], a con- 
sistent delivery order of a set of messages does not necessarily 
reflect the temporal or causal order of the events. 

The symbols used herein are defined as they are intro- 
duced. In addition, a table briefly describing each symbol is 
provided at the end of this disclosure. 

Each node is driven by an independent local physical oscil- 
lator. The oscillators of good nodes have a known bounded 
drift rate, 0^p«l, with respect to real time. For the remain- 
der of this disclosure, all references to time are with respect to 
the nominal tick, where p=0, and are simply referred to as 
clock ticks. 

Each node has two primary logical time clocks, StateTimer 
and LocalTimer, which locally keep track of the passage of 
time as indicated by the physical oscillator. In the context of 
this disclosure, all references to clock synchronization and 
self- stabilization of the system are with respect to the 
StateTimer and the LocalTimer of the nodes. There is neither 
a central clock nor an externally generated global pulse. The 
communication channels and the nodes can behave arbi- 
trarily, provided that eventually the system adheres to the 
system assumptions (see Section 2.5 below). 

FIG. 1 is a time line showing event-response delay, D, and 
network imprecision, d. The latency of interdependent com- 
munications between the nodes is expressed in terms of the 
minimum event-response delay, D, and network imprecision, 
d. As depicted in FIG. 1, a message transmitted by node N z at 
real time t 0 is expected to arrive at all destination nodes N ; . be 
processed, and subsequent messages generated by N, within 
the time interval of [t 0 +D, t 0 +D+d] for all N^€K G . Communi- 
cation between independently clocked nodes is inherently 
imprecise. The network imprecision, d, is the maximum time 
difference between all good receivers, N^., of a message from 
N z with respect to real time. The imprecision is due to the drift 
of the clocks with respect to real time, jitter, discretization 
error, temperature effects and differences in the lengths of the 
physical communication medium. These two parameters are 
assumed to be bounded such that 1 and d^O and both have 
values with units of real time nominal tick. 


1.1 Gamma (y) 

The time line is partitioned into a sequence of equally- 
spaced intervals measured by the local oscillator since the 
node transitioned into another state. Such an interval, y, is 
5 expressed in terms of the minimum event-response delay, D, 
and network imprecision, d, and is constrained such that 
y^(D+d), and is one or more local clock ticks. Therefore, the 
time-driven activities take place at equally- spaced intervals 
measured by the local oscillator since the node entered a new 
10 state. Unless stated otherwise, all time-dependent parameters 
of this protocol are measured locally and expressed as func- 
tions of y. In contrast, the event-driven activities are indepen- 
dent of y and, thus, take place immediately. 

2. Protocol Description 

15 When the system is stabilized, it is said to be in the steady 
state. In order to achieve self-stabilization, the nodes commu- 
nicate by exchanging a self-stabilization message labeled 
Sync. The Sync message is transmitted either as a result of a 
resynchronization timeout, or when a node determines that 
20 sufficient number of other nodes have engaged in the resyn- 
chronization process. 

Four fundamental parameters characterize the self-stabili- 
zation protocol, namely the topology, K, D, and d. The maxi- 
mum number of faulty nodes, F, the minimum number of 
25 good nodes, G and the remaining parameters that are subse- 
quently presented are derived parameters and are based on 
these fundamental parameters. One such derived parameter is 
y, and another is T^, which is used as a threshold in connection 
with the Sync messages. 

30 2.1 Message Validity 

Since only one self- stabilization message) namely Sync, is 
required for the proper operation of this protocol, a single 
binary value is sufficient to represent it. As a result, receiving 
such a message is indicative of its validity in the value 
35 domain. The protocol works when the timing requirements of 
the received messages from all good nodes at all other good 
nodes are not violated. The time interval between any two 
consecutive Sync messages from a node is denoted by A ss , 
and the shortest such interval is denoted by A S s,min- At the 
40 receiving nodes, the following definitions hold: 

A Sync message from a given source is valid if it arrives at 
or after A SSmin of its previous valid Sync message. 

While in the Maintain state, a Sync message from a given 
source remains valid for the duration of that state. 

45 While in the Restore state, a Sync message from a given 
source remains valid for the duration of one y. 

2.2 The Monitor 

In one embodiment, each node has a set of monitors and a 
state machine. FIG. 2 is a block diagram showing \ th node, N., 
50 with its monitors 201 etc. and state machine 202 (discussed in 
Section 2.3 below). Inputs 203 etc. come from other nodes, to 
each monitor, and output 204 goes to other nodes via broad- 
cast. 

The messages to be delivered to the destination nodes are 
55 deposited on communication channels. To closely observe 
the behavior of other nodes, a node employs (K- 1 ) monitors, 
one monitor for each source of incoming messages as shown 
in FIG. 2 . A node neither uses nor monitors its own messages . 
The distributed observation of other nodes localizes error 
60 detection of incoming messages to their corresponding moni- 
tors, and allows for modularization and distribution of the 
self-stabilization protocol process within a node. A monitor 
keeps track of the activities of its corresponding source node. 
Specifically, a monitor reads, evaluates, time stamps, vali- 
65 dates, and stores only the last valid message it receives from 
that node. A monitor maintains a logical timer, MessageT- 
imer, by incrementing it once per local clock tick. This timer 
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is reset upon receiving a valid Sync message. A monitor also 
disposes retained valid messages as appropriate. 

2.3 The State Machine 

FIG. 3 is a state transition diagram showing the node state 
machine. There are two states, Restore state 301 and Maintain 
state 302. Edge 303 relates to when a Sync message received 
in the Restore state (the system stays in the Restore state as 
explained below, because transition out of the Restore state 
depends on meeting “transitory conditions”). Edge 304 is the 
transition from the Maintain state to the Restore state, show- 
ing a Sync message being sent in that event (as explained 
below). 

The assessment results of the monitored nodes are utilized 
by the node in the self-stabilization process. The node com- 
prises a state machine and a set of (K-l) monitors. The state 
machine has two states, Restore (R) and Maintain (M), that 
reflect the current state of the node in the system as shown in 
FIG. 3. The state machine describes the behavior of the node, 
N), utilizing assessment results from its monitors, M : . . . 
M z _ l , M z+1 . . . M^ as shown in FIG. 2, where M, is the monitor 
for the corresponding node N z .. In addition to the behavior of 
its corresponding source node, a monitor’s internal status is 
influenced by the current state of the node’s state machine. 
When the state machine transitions to the Restore state the 
monitors update their internal status as appropriate. 

The transitory conditions enable the node to migrate from 
the Restore state to the Maintain state. Although during the 
self-stabilization process a node may also transition from the 
Restore state to the Maintain state upon a timeout, during 
steady state, such a time-out is indicative of an abnormal 
behavior. Therefore, the transitory conditions are defined 
with respect to the steady state where such time-outs do not 
occur. The transitory delay is the length of time a node stays 
in the Restore state. The minimum required duration for the 
transitory delay is denoted by TD min and the maximum dura- 
tion of the transitory delay by TD max . The TT> min is a derived 
parameter and a function of F. For the fully connected topol- 
ogy considered here, the transitory conditions are defined as 
follows. 

1 . The node has remained in the Restore state for at least 
TD mz>z since it entered the Restore state, where 

2, lor F-0, or 
TD mzz? =2F, for F>0, and 

2. One y has passed since the arrival of the last validSync 
message. 

The maximum duration of the transitory delay, TD max , is 
dependent on the number of additional valid Sync messages 
received and the drift rate p. The upper bound for TD max 
during steady state is given by TD max =Ap rerif0 „+(F +2) 7), 
where A Precision , also referred to as synchronization precision, 
is the guaranteed upper bound on the maximum separation 
between the LocalTimers of any two good nodes. 

In the Restore state, the node will either meet the transitory 
conditions and transition to the Maintain state, or remain in 
the Restore state for a predetermined maximum duration until 
it times out and then transition to the Maintain state. In the 
Maintain state, a node will either remain in the Maintain state 
for a predetermined maximum duration until it times out and 
transitions to the Restore state, or transition to the Restore 
state when T^ other nodes have transitioned out of the Main- 
tain state. The node transmits a Sync message when transi- 
tioning to the Restore state. 

In FIG. 4 the transitions of a good node to the Restore state 
and then from the Restore state to the Maintain state (during 
steady state) are depicted along a timeline of activities of the 
node. A Sync message is transmitted as the node transitions 
from the Restore state to the Maintain state. Activities of the 
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StateTimer and LocalTimer of the node as it transitions 
between different states are also depicted in this figure. 

Due to the inherent drift of the clocks of the good nodes, 
they need to be periodically resynchronized even if they start 
5 in perfect synchrony with respect to each other. The periodic 
synchronization during steady state is referred to as the resyn- 
chronization process, whereby all good nodes transition to the 
Restore state and then synchronously to the Maintain state. 
The resynchronization process begins when the first good 
10 node transitions to the Restore state and ends after the last 
good node transitions to the Maintain state. 

The synchronization period is defined as the maximum 
time interval (during steady state) that a good node engages in 
the resynchronization process. The synchronization period 
15 depends on the maximum durations of both states of the 
node’s state machine. The maximum duration for the Restore 
state is denoted by p*. and the maximum duration for the 
Maintain state is denoted by Pmj where p* and are 
expressed in terms of y. The length of time a good node stays 
20 in the Restore state is denoted by L^. During steady state 
is always less than P^. The length of time a good node stays in 
the Maintain state is denoted by L M . The effective synchro- 
nization period, P ActuaI , is the time interval (during steady 
state) between the last two consecutive resets of the Local - 
25 Timer of a good node in a stabilized system, where 
^Actual Lr+L^P^ +p m . 

The time interval between any two consecutive Sync mes- 
sages from a node is denoted by A^. The shortest such inter- 
val is denoted by A S s,min> an d it follows that A SSmin = 
30 (TD mzw *y+l) clock ticks. 

A node keeps track of time by incrementing its logical time 
clock StateTimer once every y. After the StateTimer reaches 
P^ or P^ depending on the current state of the node, the node 
times out, resets the StateTimer, and transitions to the other 
35 state. If the node was in the Maintain state it transmits a new 
Sync message, The current value of this timer reflects the 
duration of the current state of the node. 

This protocol does not maintain a history of past behavior 
of the nodes. All such determinations about the health status 
40 of the nodes in the system are assumed to be done by higher 
level mechanisms. 

This protocol is expected to be used as the fundamental 
mechanism in bringing and maintaining a system within a 
known synchronization precision bound. Therefore, the pro- 
45 tocol has to properly filter out inherent oscillations in the 
StateTimer during the resynchronization process as depicted 
in FIG. 4. This issue is resolved by using the LocalTimer in 
the protocol. The logical time clock LocalTimer is incre- 
mented once every local clock tick and is reset either when it 
50 reaches its maximum allowed value or when the node has 
transitioned to the Maintain state and remained in that state 
for ResetLocalTimerAt local clock ticks, where ResetLocal- 
TimerAt is constrained by the following inequality: 

55 C kpredsioJy] = ResetLocalTimerAt^/ 3 ^ [ A Precision /y] (1) 

ResetLocalTimerAt can be given any value in its range as 
specified in inequality (1 ). However, its specific value must be 
the same at all good nodes. We chose the earliest such value, 
ResetLocalTimerAt=[A / , recz - yzo „/yl, to reset the LocalTimer of 
60 all good nodes. Any value greater than [ A Precisio Jy] will 
prolong the convergence time. 

The LocalTimer is intended to be used by higher level 
protocols and must be managed properly to provide the 
desired behavior. The LocalTimer is also used in assessing the 
65 state of the system in the resynchronization process and is 
bounded by P, where P=P i? +P Af . During stead state, the value 
of LocalTimer is always less than P. 



7 


US 8,255,732 B2 


2.4 Protocol Functions 

The functions used in this protocol are described in this 
section. 

The function InvalidSync( ) is used by the monitors. This 
function determines whether a received Sync message is 5 
invalid. When this function returns a true value, it indicates 
that an unexpected behavior by the corresponding source 
node has been detected. 

The function ConsumeMessage( ) is used by the monitors. 
When the host node is in the Restore state, the monitor invali- 
dates the stored Sync message after it has been kept for y. 

The Retry( ) function determines if at least T^ other nodes 
have transitioned out of the Maintain state, where T i? =F+l. 
When at least T^ valid Sync messages from as many nodes 15 
have been received, this function returns a true value indicat- 
ing that at least one good node has transitioned to the Restore 
state. This function is used to transition from the Maintain 
state to the Restore state. 

The TransitoryConditionsMet( ) function determines 20 
proper timing of the transition from the Restore state to the 
Maintain state. This function keeps track of the passage of 
time by monitoring StateTimer and determines if the node has 
been in the Restore state for at least TD min . It returns a true 
value when the transitory conditions are met. 25 

The TimeOutRestore( ) function uses p* as a boundary 
value and asserts a timeout condition when the value of the 
StateTimer has reached P A . Such timeout triggers the node to 
transition to the Maintain state. 

The TimeOutMaintain( ) function uses as a boundary 30 
value and asserts a timeout condition when the value of the 
StateTimer has reached V M . Such timeout triggers the node to 
reengage in another round of synchronization. This function 
is used when the node is in the Maintain state. 

In addition to the above functions, the state machine uti- 35 
lizes the TimeoutGammaTimer( ) function. This function is 
used to regulate node activities at the y boundaries. It main- 
tains a GammaTimer by incrementing it once per local clock 
tick and once it reaches the duration of y, it is reset and the 
function returns a true value. 40 

2.5 System Assumptions 

The system assumptions are defined as follows. 

1 . The cause of transient faults has dissipated. 

2. All good nodes actively participate in the self-stabiliza- 

tion process and correctly execute the protocol. 45 

3. At most F of the nodes remain faulty. 

4. The source of a message is distinctly identifiable by the 
receivers from other sources of messages. 

5. A message sent by a good node will be received and 
processed by all other good nodes within y, where 50 
y^(D+d). 

6. The initial values of the state and all variables of a node 
can be set to any arbitrary value within their correspond- 
ing range. (In an implementation, it is expected that 
some local capabilities exist to enforce type consistency 55 
for all variables.) 

2.6 The Self-Stabilizing Clock Synchronization Problem 

To simplify the presentation of this protocol, it is assumed 

that all time references are with respect to an initial real time 
to, where t o =0 when the system assumptions are satisfied, and 60 
for all t>t 0 the system operates within the system assump- 
tions. Let 

C be the bound on the maximum convergence time, 

^LocaiTimeJ^)^ f° r real time t, be the maximum difference of 
values of the local timers of any two good nodes N z and 65 
N ; , where N z -, N^eK^, and K G is the set of all good nodes, 
and 
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A precision j a l so referred to as self- stabilization or synchro- 
nization precision, be the guaranteed upper bound on the 
maximum separation between the local timers of any 
two good nodes N f and N ; in the presence of a maximum 
of F faulty nodes, where N z , N^K^. 

A good node N f resets its variable LocalTimer z periodically 
but at different points in time than other good nodes. The 
difference of local timers of all good nodes at time t, A Locat 
Timer(t), is determined by the following equation while recog- 
nizing the variations in the values of the Local Timer z - across 
all good nodes. 

Ai o ^m rae r(0 = mm((LocalTimer max (r)-LocalTimer wz>1 
(?)), (LocalTimer md3X (r-r)-LocalTimer m2>J (?-r))), 

where, 

I=\^ Pn! cnsioJ Yl> 

LocalTimer mzw (x)=min(LocalTimer z (x)), 
LocalTimer wz2X (x)=max(LocalTimer.(x)), and 
there exist C and k precision such that: 

1. Convergence: b LocalTimer {i)^ Precisio „ 

2. Closure: VtgC, A Lo cammer$)=bprectsi<m 

3. Congruence: VN Z , N^€K G , Vti^C, LocalTimer z (t)=0— >N Z 
and Nj are in the Maintain state. 

The values of C, A p recision (after an elapsed time of P), and 
the maximum value for LocalTimer z , P, are determined to be: 

C=(2P R +P M )-y 
^ Precision =(3F-iyy-D+A Drift , 

P = Pr + Pm> 

Pm >:> Pr’ 

where the amount of drift from the initial precision is given by 

Az>,/r((i+p)-i/(i+p))^r 

Note that since P>( 1 A)P jR and since the LocalTimer is reset 
after reaching P (worst case wraparound), a trivial solution is 
not possible. 

3. A Self- Stabilizing Byzantine-Fault-Tolerant Clock Syn- 
chronization Protocol 

The presented protocol is described in FIG. 5 and com- 
prises a state machine and a set of monitors that execute once 
every local oscillator tick. 

The semantics of the pseudocode in FIG. 5 are as follows: 
Indentation is used to show a block of sequential state- 
ments. 

Commas (,) are used to separate sequential statements. 

A period (.) is used to end a statement. 

A period combined with a comma (.,) is used to mark the 
end of a statement and at the same time to separate it 
from other sequential statements. 

The operational steps for each monitor 501 are: 

1 . if there is an incoming message from the node corre- 
sponding to the monitor: 

(a) determining if the message is a valid Sync message; 

(b) if the message is a valid Sync message, validating 
and storing the message; 

(c) if the message is not a valid Sync message, invalidat- 
ing the message; 

2. otherwise, if there is no such message, doing nothing 
With regard to the state machine 502, for the Restore state 

503, the protocol steps are: 

1 . determining if the node has timed out in the Restore 
state; 

2. if the node has timed out in the Restore state, 
resetting the StateTimer; and 
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changing the machine state for the node to the Maintain 
state; 

3. if the node has not timed out in the Restore state, 
determining if transitory conditions are met (i.e., that (a) 

the node has remained in the Restore state, since last 
entering that state, for a period equal to or greater than 
two StateTimer ticks, where the number of said faulty 
nodes is zero, or two times the number of faulty nodes, 
where the number of faulty nodes is greater than zero, 
and (b) a period of at least one y has passed since the 
arrival of the last valid Sync message); 
if the transitory conditions are met, 
resetting the StateTimer; 

changing the machine state for the node to the Main- 
tain state; 

if the transitory conditions are not met, 

keeping the machine state for the node in the Restore 
state; 

4. else (if the node has not timed out in the Restore state), 
keeping the machine state for the node in the Restore 

state; 

For the Maintain state 504, the protocol steps are: 

1 . if either (a) the StateTimer has exceeded or (b) the 
number of valid Sync messages received from other 
nodes is at least one more than the number of faulty 
nodes, 

(i) broadcasting a Sync message to all of said other 
nodes; 

(ii) resetting the StateTimer clock; 

(iii) changing the machine state for the node to the 
Restore state; 

2. else (if neither (a) the StateTimer has exceeded Pat? nor 
(b) the number of valid Sync messages received from 
other nodes is at least one more than the number of faulty 
nodes, and (c) the GammaTimer clock has reached the 
duration of y), 

(a) if the value of the StateTimer clock equals [ A p recisio J 
y], resetting said LocalTimer clock; 

(b) keeping the machine state for the node in the Main- 
tain state; 

3. else (if neither (a) the StateTimer has exceeded a prede- 
termined maximum interval, nor (b) the number of valid 
Sync messages received from other nodes is at least one 
more than the number of faulty nodes, and (c) the Gam- 
maTimer clock has not reached the duration of y), 
keeping the machine state for the node in the Maintain 

state. 

To avoid introducing oscillations in the system, has to be 

sufficiently large to allow time to reset the LocalTimer after 
the node transitions to the Maintain state. In other words, 
i> a>A / v C c, j , 0 „+ i -atest to Maintain state (lM)+b Preciston . 


In general, and for all F>0 and K^3F+1, and to prevent an 
early timeout, must be constrained in accordance with the 

previous paragraph. The maximum duration for the Maintain 
state, Pat? is typically much larger than P^. Thus, P^is derived 


Since this protocol self-stabilizes from any state, initializa- 
tion and/or reintegration are not treated as special cases. 
Therefore, a reintegrating node will always be admitted to 
participate in the self-stabilization process as soon as it 
5 becomes active. 

Since ^ Ac tuaf ^ m an d typically P^is much greater than 
P^ the maximum convergence time, C, can be approximated 
to C~P. Therefore, C is a linear function of P, and, similarly, C 

is a linear function of P^. 
to 

A model of this protocol has been mechanically verified 
using the SMV state machine language where the entire state 
space is examined, and proven to self-stabilize in the presence 
of one arbitrary faulty node. 

15 4. Protocol Overhead 

Since only one message, namely Sync, is required for the 
operation of this protocol, therefore, during steady state the 
protocol overhead is at most (depending on the amount of 
A Drift) two messages per P. Also, since only one message is 
20 needed, a single binary value is sufficient to represent it. 

5. Applications 

The self-stabilizing protocol disclosed herein has many 
practical applications. Embedded systems, distributed pro- 
cess control, synchronization, inherent fault tolerance which 
25 also includes Byzantine agreement, computer networks, the 
Internet, Internet applications, security, safety, automotive, 
aircraft, wired and wireless telecommunications, graph theo- 
retic problems, leader election, and time division multiple 
access (TDMA), are a few examples. These are some of the 
30 many areas of distributed systems that can use self-stabiliza- 
tion in order to design more robust distributed systems. 

6. Achieving Tighter Precision 

Since the time-driven self-stabilization activities take place 
at y intervals, if y, and hence ^ recisio „, are larger than the 
35 desired precision, the system is said to be coarsely synchro- 
nized. Otherwise, the system is said to be finely synchronized. 
If the granularity provided by the self-stabilization precision 
is coarser than desired, a higher synchronization precision 
can be achieved in a two step process. First, a system from any 
40 initial state has to be coarsely synchronized and guaranteed 
that the system remains coarsely synchronized and operates 
within a known precision, A Precision - The second step, in con- 
junction with the coarse synchronization protocol, is to utilize 
a proven protocol that is based on the initial synchrony 
45 assumptions to achieve optimum precision of the synchro- 
nized system as depicted in FIG. 6. 

As depicted in FIG. 6, the coarse synchronization protocol 
601 initiates the start of the fine synchronization protocol 603 
if a tighter precision of the system is desired (602). The coarse 
50 synchronization protocol maintains self-stabilization of the 
system while the fine synchronization protocol increases the 
precision of the system. 

The necessary conditions to initiate the fine synchroniza- 
tion protocol are that convergence has to be achieved and all 
55 good nodes have to be in the Maintain state. It follows from 
Theorem Congruence that upon convergence all good nodes 
are in the Maintain state. Thus, examination of the current 
state as well as the value of the StateTimer of the good nodes 
provides the necessary conditions to attempt to initiate the 
60 fine synchronization protocol. 

It is apparent, based on the foregoing, that the invention 
meets the objectives set forth above. Although the invention 
has been described in detail, it should be understood that 
various changes, substitutions, and alterations may be readily 
65 ascertainable by those skilled in the art and may be made 
herein without departing from the spirit and scope of the 
present invention as defined by the claims appended hereto. 
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The following table lists the symbols used in the protocol. 


Symbols 

Description 

P 

bounded drift rate with respect to real time 

d 

network imprecision 

D 

event-response delay 

F 

maximum number of faulty nodes 

G 

minimum number of good nodes 

K 

sum of all nodes 

K * 

set of all good nodes 

Sync 

self-stabilization message 

S 

abbreviation for Sync message 

&ss 

time difference between the last consecutive Sync 
messages 

T * 

threshold for Retry( ) function 

Restore 

self- stabilization state 

Maintain 

self- stabilization state 

R 

abbreviation for Restore state 

M 

abbreviation for Maintain state 

u 

maximum duration while in the Restore state 


minimum value of 

Pm 

maximum duration while in the Maintain state 

^Actual 

effective synchronization period 

V 

equal space time intervals for time-driven 
activities 

C 

maximum convergence time 

^LocalTimerO) 

maximum time difference of LocalTimers of any two 
good nodes at real time t 

LM 

Latest Maintain 

EM 

Earliest Maintain 

AzMEM 

difference of LM and EM, initial self-stabilization 
precision 

^ Precision 

maximum self-stabilization precision 

A Drift 

maximum deviation from the initial synchrony 

N i 

the i* node 

M. 

the \ th monitor of a node 


What is claimed as new and desired to be secured by 35 
Letters Patent of the United States is: 

1. A method for synchronizing clocks among a plurality of 
nodes in a system, the plurality of nodes comprising faulty 
nodes and good nodes having an arbitrary state and in the 4Q 
presence of a bounded number of arbitrary faults, said nodes 
being capable of communicating with each other by exchang- 
ing messages through a set of communication channels, com- 
prising performing the following steps at each node: 

( 1 . 1 ) providing a local oscillator clock which provides a 45 
local oscillator clock tick; 

( 1 . 2 ) executing once every local oscillator clock tick (a) a 
plurality of operational steps for each of a set of monitors 
comprising one monitor for and corresponding to each 
other node in said plurality of nodes, and (b) a plurality 50 
of protocol steps for a state machine for each node, 
wherein the state machine comprises a Restore state and 

a Maintain state; 

(1.3) providing a GammaTimer clock and incrementing 
said GammaTimer clock once per local oscillator clock 55 
tick; 

(1 .4) providing a StateTimer clock, and incrementing said 
StateTimer clock on each equally spaced interval y com- 
prising one or more local oscillator clock ticks such that 

y is equal to or greater than the sum of a minimum event 60 
response delay D among said plurality of nodes and a 
network imprecision d; and 

(1.5) providing a LocalTimer clock, and incrementing said 
LocalTimer clock on the local oscillator clock tick. 

2. The method of claim 1, wherein said plurality of opera- 65 
tional steps for a particular monitor of said set of monitors 
comprises: 
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(2.1) determining if upon the local oscillator clock tick 
there is an incoming message from the node correspond- 
ing to said particular monitor; 

(2.2) if there is such an incoming message, 

(2.2.1) determining if said message is a valid Sync mes- 
sage, wherein said message is considered valid if the 
time between its arrival and the arrival of a prior Sync 
message is equal to or greater than an interval 
regarded as the shortest permissible interval between 
any two consecutive Sync messages; 

(2.2.2) if said message is a valid Sync message, validat- 
ing and storing said message; and 

(2.2.3) if said message is not a valid Sync message, 
invalidating said message; 

(2.3) otherwise, if there is no message, doing nothing. 

3. The method of claim 2, wherein said steps of said state 
machine comprises determining the state machine for the 
node. 

4. The method of claim 3, wherein: 

(4.1) if the state machine is in the Restore state, 

(4.1.1) determining if the node has timed out in the 
Restore state by exceeding a limit time duration that 
has been selected for the Restore state; 

(4.1.2) if the node has timed out in the Restore state, 

(4. 1.2.1) resetting the StateTimer; and 

(4. 1.2. 2) changing the state machine for the node to 
the Maintain state; 

(4.1.3) if the node has not timed out in the Restore state, 

(4. 1 .3. 1) determining if transitory conditions are met, 
said transitory conditions comprising the condi- 
tions that (a) the node has remained in the Restore 
state, since last entering that state, for a period 
equal to or greater than two StateTimer ticks when 
a number of said faulty nodes is zero, or two times 
the number of faulty nodes, when the number of 
said faulty nodes is greater than zero, and (b) a 
period of at least one y has passed since the arrival 
of the last valid Sync message; 

(4. 1.3. 2) if the transitory conditions are met, 
resetting the StateTimer, and changing the state 

machine for the node to the Maintain state; 

(4. 1 .3.3) if the transitory conditions are not met, keep- 
ing the state machine for the node in the Restore 
state; 

(4.1.4) if the node has not timed out in the Restore state, 

(4. 1.4.1) keeping the state machine for the node in the 
Restore state; 

(4.2) if the state machine is in the Maintain state, 

(4.2.1) determining if either (a) the StateTimer has timed 
out by exceeding a limit time duration selected for the 
Maintain state, or (b) the number of valid Sync mes- 
sages received from other nodes is at least one more 
than the number of faulty nodes, 

(4.2.2) if either (a) the StateTimer has timed out, or (b) 
the number of valid Sync messages received from 
other nodes is at least one more than the number of 
faulty nodes, 

(4.2.2. 1) broadcasting a Sync message to all of said 
other nodes; 

(4.2.2.2) resetting said StateTimer clock; and 

(4.2.4.3) changing the state machine for the node to 
the Restore state; 

(4.2.3) if neither (a) the StateTimer has timed out, nor (b) 
the number of valid Sync messages received from 
other nodes is at least one more than the number of 
faulty nodes, but if the GammaTimer clock has 
reached the duration of y, 
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(4.23.1) determining if the value of the StateTimer 

clock is equal to or greater than the next higher 
integer value of the quotient of an upper bound on 
the maximum separation between the LocalTimer 
clocks of any two good nodes, divided by y; 5 

(4.23.2) if the value of the StateTimer clock is equal 
to or greater than said next higher integer value, 
resetting said LocalTimer clock; 

(4 .2 3 3) keeping the state machine for the node in the 
Maintain state; 10 

(4.2.4) if neither (a) the StateTimer has timed out, nor (b) 
the number of valid Sync messages received from 
other nodes is at least one more than the number of 
faulty nodes, and (c) the GammaTimer clock has not 
reached the duration of y, 15 

(4 .2 .4 . 1 ) keeping the state machine for the node in the 
Maintain state. 

5. The method of claim 4, wherein said limit time duration 

for the Restore state, P^, is chosen such that (a) if 0 D, 

then P^>7F-1, (b) if k Drift =D, then P J? >7F+1, and (e) if 20 
2D>A Z) ^>D, then P^>7F+3, where F is the number of faulty 
nodes, A Drift , a maximum deviation from the initial syn- 
chrony, is ((1 +p)— 1/(1 +p))P*y, p is a drift rate (0^p«l) of 
the local oscillator clock, and P, an effective synchronization 
period, is a time interval between the last two consecutive 25 
resets of the LocalTimer, measured at a steady state when said 
system is stabilized. 

6. The method of claim 5, wherein an interval regarded as 
the shortest permissible interval between any two consecutive 
Sync messages, A SSm ,.„, is (TD m ,.„+l), where TD„ m is 2 for 30 
cases when the number of faulty nodes is zero, otherwise 
TD mzw is equal to two times the number of faulty nodes when 
the number of faulty nodes is greater than zero. 

7. The method of claim 5, wherein said method requires 

only one message, which message is Sync. 35 

8. The method of claim 5, wherein a synchronization pre- 
cision, A p recisiori , comprising the upper bound on the maxi- 
mum separation between the LocalTimers of any two good 
nodes, equals 

40 

(3 F-l)-y-D + A Drift . 

9. The method of claim 8, wherein a bound on the maxi- 
mum convergence time, C, equals ^P^+P^-y wherein Pat is 
a maximum duration for the Maintain state. 

10. The method of claim 9, wherein the P, is a maximum 45 
time interval between two consecutive resets of the Local- 
Timer by a good node, and is equal to P^+P^ during said 
steady state. 

11. The method of claim 10, wherein upon reaching said 
steady state when said system is stabilized, the following 50 
properties are obtained: (a) convergence, wherein for all val- 
ues of t greater than or equal to C, the maximum difference of 
values of the LocalTimers of all good nodes in said plurality 
of nodes. k LocaITimer (t\ is less than or equal to k Precisio „, (b) 
closure, wherein for all values of t greater than C it remains 55 
true that A Locatzlmer (t) is less than or equal to A Precisio „, and (c) 
congruence, wherein for all said good nodes, and for all 
values of t greater than C, the condition that LocalTimer(t)=0 
for any such good node implies that all of said good nodes are 

in the Maintain state. 60 

12. The method of claim 5, wherein said limit me duration 
for the Maintain state is chosen to be a value greater than the 
limit time duration for the Restore state. 

13 . The method of claim 5, further comprising determining 

if all good nodes are in the Maintain state by examining the 65 
current state of said good nodes and the StateTimer value of 
said good nodes, and if all good nodes are in the Maintain 
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state, performing a fine synchronization by reducing the 
upper bound on the maximum separation between the Local - 
Timer clocks of any two good nodes. 

14. The method of claim 1, wherein said system does not 
comprise a central clock used by said nodes for synchroniza- 
tion. 

15. The method of claim 1, wherein said nodes do not use 
an externally generated global pulse. 

16. A clock synchronization control element for a node 
within a plurality of nodes in a system comprising faulty 
nodes and good nodes, having an arbitrary state and in the 
presence of a bounded number of arbitrary faults, said nodes 
being capable of communicating with each other by exchang- 
ing messages through a set of communication channels, com- 
prising: 

(16.1) a local oscillator clock having a local oscillator 
clock tick; 

(16.2) a set of monitors comprising one monitor for and 
corresponding to each other node in said plurality of 
nodes, and a state machine for the node, having a Restore 
state and a Maintain state, said monitors and said state 
machine each being executed on each tick of said local 
oscillator clock; 

(163) a StateTimer clock, incremented on each equally 
spaced interval y comprising one or more local oscillator 
clock ticks such that y is equal to or greater than the sum 
of a minimum event response delay D among said plu- 
rality of nodes and a network imprecision d; 

(16.4) a LocalTimer clock, incremented on every local 
oscillator clock tick; and 

(16.5) a GammaTimer clock, incremented once per local 
oscillator clock tick. 

17. The control element in accordance with claim 16, 
wherein a monitor in the set of monitors comprises logic to 
determine if upon the local oscillator clock tick there is an 
incoming message from the node corresponding to said moni- 
tor, and if there is an incoming message, to determine if said 
incoming message is a valid Sync message by testing if a time 
comprising the difference between an arrival time and an 
arrival time of a prior Sync message is equal to or greater than 
an interval regarded as the shortest permissible interval 
between any two consecutive Sync messages, and if said 
incoming message is a valid Sync message, to validate and 
store said incoming message, and if said incoming message is 
not a valid Sync message, to invalidate said incoming mes- 
sage, and otherwise, if there is no incoming message, to do 
nothing. 

18. The control element in accordance with claim 16, 
wherein said state machine comprises logic to determine a 
state of the state machine for the node. 

19. The control element in accordance with claim 18, 
wherein said state machine comprises logic executable in said 
Restore state to determine if the node has timed out in the 
Restore state by testing whether said node has been in the 
Restore state for more than a selected limit time duration for 
the Restore state, and if the node has timed out in the Restore 
state, to reset the StateTimer, and change the state machine for 
the node to the Maintain state, and if the node has not timed 
out in the Restore state, determining if transitory conditions 
are met, said transitory conditions comprising that (a) the 
node has remained in the Restore state, since last entering that 
state, for a period equal to or greater than two StateTimer 
ticks, when the number of said faulty nodes is zero, or two 
times the number of faulty nodes, when the number of said 
faulty nodes is greater than zero, and (b) a period of at least 
one y has passed since an arrival of a last valid Sync message, 
and if the transitory conditions are met, to reset the 
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StateTimer, and change the state machine for the node to the 
Maintain state, and, if the transitory conditions are not met, to 
keep the state machine for the node in the Restore state, and 
otherwise, if the node has not timed out in the Restore state, to 
keeping the state machine for the node in the Restore state. 

20. The control element in accordance with claim 19, 
wherein said state machine comprises logic executable in said 
Maintain state to determine if either (a) the StateTimer has 
exceeded a predetermined maximum interval, or (b) a number 
of valid Sync messages received from other nodes is at least 
one more than the number of faulty nodes, and either (a) the 
StateTimer has exceeded a predetermined maximum interval, 
or (b) the number of valid Sync messages received from other 
nodes is at least one more than the number of faulty nodes, to 
broadcast a Sync message to all of said other nodes, reset said 
StateTimer clock, and change the state machine for the node 
to the Restore state, and if neither (a) the StateTimer has 
exceeded a predetermined maximum interval, nor (b) the 
number of valid Sync messages received from other nodes is 
at least one more than the number of faulty nodes, but if the 
GammaTimer clock has reached the duration of y, to deter- 
mine if the value of the StateTimer clock is equal to or greater 
than the next higher integer value of the of an upper bound on 
the maximum separation between the LocalTimer clocks of 
any two good nodes, divided by y, and if the value of the 
StateTimer clock is equal to or greater than said next higher 
integer value, to reset said LocalTimer clock, to keep the state 
machine for the node in the Maintain state, and otherwise, if 
neither (a) the StateTimer has exceeded the predetermined 
maximum interval, nor (b) the number of valid Sync mes- 
sages received from other nodes is at least one more than the 
number of faulty nodes, and (c) the GammaTimer clock has 
not reached the duration of y , to keep the state machine for the 
node in the Maintain state. 

21. The control element in accordance with claim 20, 
wherein said limit time duration for the Restore state, P^, is 
chosen such that (a) if 0^A Drt ^<D, then P i? >7F-l, (b) if 
^Dnff I), th en Pjg>7F+l, and (c) if 2D>A Dri yf>D, then 
P^>7F+3, where F is the number of faulty nodes, A Dri ^ a 
maximum deviation from the initial synchrony, is ((l+p)-l/ 
(l+p))P y, p is a drift rate (0^p«l) of the local oscillator 
clock, and P, an effective synchronization period, is a time 
interval between the last two consecutive resets of the Local- 
Timer, measured at a steady state when said system is stabi- 
lized. 

22. The control element in accordance with claim 20, 
wherein an interval regarded as the shortest permissible inter- 
val between any two consecutive Sync messages, A SS min , is 
(TD miw *y+l), where TD mzw is 2 for cases when the number of 
faulty nodes is zero, and otherwise TT> min is equal to two times 
the number of faulty nodes when the number of faulty nodes 
is greater than zero. 

23. The control element in accordance with claim 20 
wherein said node requires only one message, which message 
is Sync. 

24. The control element in accordance with claim 20, 
wherein a synchronization precision, A Precision , which is the 
upper bound on the maximum separation between the Local- 
Timers of any two good nodes, equals (3F-l)*y-D+A £>rz - /? . 

25. The control element in accordance with claim 24, 
wherein a bound on the maximum convergence time, C, 
equals (2P R +P M ) y wherein P^is a maximum duration for the 
Maintain state. 

26. The control element in accordance with claim 25, 
wherein the P is a maximum time interval between two con- 
secutive resets of the LocalTimer by a good node, and is equal 
to P^+P^ during said steady state. 
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27. The control element in accordance with claim 16, 
wherein said system does not comprise a central clock used 
by said nodes for synchronization. 

28. The control element in accordance with claim 16, 
5 wherein said nodes do not use an externally generated global 

pulse. 

29. The control element in accordance with claim 26, 
wherein upon reaching said steady state when said system is 
stabilized, the following properties are obtained: (a) conver- 

10 gence, wherein for all values of t greater than or equal to C, a 
maximum difference of values of the local timers of any good 
node in said plurality of nodes, A LocalIimer {X ), is less than or 
equal to A Precision , (b) closure, wherein for all values of t 
1 5 greater than C it remains true that A LocalTimer (X) is less than or 
equal to A Precision , and (c) congruence, wherein for all said 
good nodes, and for all values of t greater than C, the condi- 
tion that LocalTimer(t)=0 for any such good node implies that 
all of said good, nodes are in the Maintain state. 

20 30. The control element in accordance with claim 20, 

wherein said predetermined maximum interval for said Main- 
tain state is chosen to be a value greater than the limit time 
duration for the Restore state. 

31. The control element in accordance with claim 20, com- 
25 prising logic to determine if all good nodes are in the Maintain 

state by examining the current state of said node and the 
StateTimer value of said good nodes, and if all good nodes are 
in the Maintain state, to perforin a fine synchronization by 
reducing the upper bound on the maximum separation 
30 between the LocalTimer clocks of any two good nodes. 

32. A clock synchronization control element for a node 
within a plurality of nodes in a system, having an arbitrary 
state and in the presence of a bounded number of arbitrary 

35 faults, said nodes being capable of communicating with each 
other by exchanging messages through a set of communica- 
tion channels and said nodes comprising good nodes and 
faulty nodes, comprising: 

(32.1) a local oscillator clock having a local oscillator 
40 clock tick; 

(32.2) a set of monitors comprising one monitor for and 
corresponding to each other node in said plurality of 
nodes, and a state machine for the node, having a Restore 
state and a Maintain state, said monitors and said state 

45 machine each being executed on each tick of said local 
oscillator clock; 

(32.3) a StateTimer clock, incremented on each equally 
spaced interval y comprising one or more local oscillator 
clock ticks such that y is equal to or greater than the sum 

50 of the minimum event response delay D among said 
plurality of nodes and the network imprecision d; 

(32.4) a LocalTimer clock, incremented on every local 
oscillator clock tick; 

(32.5) a GammaTimer clock and incrementing said Gam- 
55 maTimer clock once per local oscillator clock tick; 

(32.6) a monitor in the set of monitors comprising logic to 
determine if upon the local oscillator clock tick there is 
an incoming message from the node corresponding to 
said monitor, and if there is an incoming message, to 

60 determine if said incoming message is a valid Sync 
message by testing if a time between an arrival time and 
an arrival time of a prior Sync message is equal to or 
greater than an interval comprising a shortest permis- 
sible interval between any two consecutive Sync mes- 
65 sages, and if said incoming message is a valid Sync 
message, to validate and store said incoming message, 
and if said incoming message is not a valid Sync mes- 
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sage, to invalidate said incoming message, and other- 
wise, if there is no such incoming message, to do noth- 
ing; 

(32.7) said state machine comprising: 

(32.7.1) logic to determine a state of the state machine 5 
for the node; 

(32.7.2) logic executable in said Restore state to deter- 
mine if the node has timed out in the Restore state by 
testing whether said node has been in the Restore state 
for more than a selected limit time duration for the 
Restore state, and if the node has timed out in the 
Restore state, to reset the StateTimer, and change the 
state machine for the node to the Maintain state, and if 
the node has not timed out in the Restore state, deter- 15 
mining if transitory conditions are met, said transitory 
conditions comprising that (a) the node has remained 

in the Restore state, since last entering that state, for a 
period equal to or greater than two StateTimer ticks 
when a number of said faulty nodes is zero, or two 20 
times the number of faulty nodes, when the number of 
said faulty nodes is greater than zero, and (b) a period 
of at least one y has passed since the arrival of the last 
valid Sync message, and if the transitory conditions 
are met, to reset the StateTimer, and change the state 25 
machine for the node to the Maintain state, and, if the 
transitory conditions are not met, to keep the state 
machine for the node in the Restore state, and other- 
wise, if the node has not timed out in the Restore state, 
to keeping the state machine for the node in the 30 
Restore state; 

(32.7.3) logic executable in said Maintain state to deter- 
mine if either (a) the StateTimer has exceeded a pre- 
determined maximum interval, or (b) the number of 
valid Sync messages received from other nodes is at 35 
least one more than the number of faulty nodes, and 
either (a) the StateTimer has exceeded the predeter- 
mined maximum interval, or (b) the number of valid 
Sync messages received from other nodes is at least 
one more than the number of faulty nodes, to broad- 40 
cast a Sync message to all of said other nodes, reset 
said StateTimer clock, and change the state machine 
for the node to the Restore state, and if neither (a) the 
StateTimer has exceeded the predetermined maxi- 
mum interval, nor (b) the number of valid Sync mes- 45 
sages received from other nodes is at least one more 
than the number of faulty nodes, but if the Gamma- 
Timer clock has reached the duration of y, to deter- 
mine if the value of the StateTimer clock is equal to or 
greater than a next higher integer value of the of upper 50 
bound on the maximum separation between the 
LocalTimer clocks of any two good nodes, divided by 

y, and if the value of the StateTimer clock is equal to 
or greater than said next higher integer value, to reset 
said LocalTimer clock, to keep the state machine for 55 
the node in the Maintain state, and otherwise, if nei- 
ther (a) the StateTimer has exceeded the predeter- 
mined maximum interval, nor (b) the number of valid 
Sync messages received from other nodes is at least 
one more than the number of faulty nodes, and (c) the 
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GammaTimer clock has not reached the duration of y, 
to keep the state machine for the node in the Maintain 
state; 

(32.8) wherein said limit time duration for the Restore 
state, is chosen such that (a) if 0^A Drift <D 7 then 
P*>7F-1, (b) if A nrif =D, then P*>7F+1, and (c) if 
2T»A Drif p>T>, then P jR > 7F+3, where F is the number of 
faulty nodes, A Drift , a maximum deviation from the ini- 
tial synchrony, is ((l+p)-l/(l+p))P-y, p is the drift rate 
(0^p«l) of the local oscillator clock, and P an effec- 
tive synchronization period, is a time interval between 
the last two consecutive resets of the LocalTimer, mea- 
sured at the steady state when said system is stabilized; 

(32.9) wherein an interval regarded as the shortest permis- 
sible interval between any two consecutive Sync mes- 
sages, A SS mi „, is (TD mlM v+l), where TD ml „ is 2 for cases 
when the number of faulty nodes is zero, and otherwise 
TY> min is equal to two times the number of faulty nodes 
when the number of faulty nodes is greater than zero; 

(32.10) wherein said node requires only one message, 
which message is Sync; 

(32.11) wherein the synchronization precision, A Precision , 
which is the upper bound on the maximum separation 
between the LocalTimers of any two good nodes, equals 
OF-lb-D+A^; 

(32.12) wherein a bound on the maximum convergence 
time, C, equals (2P i? +P^ r )*y wherein is a maximum 
duration for the Maintain state; 

(32.13) wherein P, is a maximum time interval (during 
steady state) between two consecutive resets of the 
LocalTimer by a good node, and is equal to P^+P^; 

(32.14) wherein said system does not comprise a central 
clock used by said nodes for synchronization; 

(32.15) wherein said nodes do not use an externally gen- 
erated global pulse; and 

(32.16) such that, upon reaching said steady state when 
said system is stabilized, the following properties are 
obtained: (a) convergence, wherein after all times for all 
values of t greater than or equal to C, the maximum 
difference of values of the local timers of any good node 
in said plurality of nodes, A LocalTimer (X ), is less than or 
equal to A Precision , (b) closure, wherein for all values of 
t greater than C it remains true that A LocalTimer {\) is less 
than or equal to A Precision , and (c) congruence, wherein 
for all said good nodes, and for all values of t greater than 
C, the condition that LocalTimer(t)=0 for any such good 
node implies that all of said good nodes are in the Main- 
tain state. 

33. A node in accordance with claim 32, wherein said 
predetermined maximum interval for said Maintain state is 
chosen to be a value greater than the limit time duration for the 
Restore state. 

34. A node in accordance with claim 32, further comprising 
logic to determine if all good nodes are in the Maintain state 
by examining the current state of said node and the 
StateTimer value of said good nodes, and if all good nodes are 
in the Maintain state, to perform a fine synchronization by 
reducing the upper bound on the maximum separation 
between the LocalTimer clocks of any two good nodes. 



